Method and system for facilitating wireless, full-body, real-time user interaction with digitally generated text data

ABSTRACT

An interactive display system includes an electronic camera which intermittently records a participant&#39;s image and digitally inputs the video image into a computer, creating a digitized “virtual user image”. The camera is positioned behind a large viewing screen which faces the user and records the user&#39;s image through an opening provided within the screen. The computer further receives a text string having a fall rate value. For each digital image received by the camera, a software program determines the destination of each line of text according to its fall rate, and then uses pixel color comparison techniques to determine if a “virtual obstacle” is located at the particular destination. If not, the text is displayed at the destination so that the text appears to “fall” on the screen. If a virtual obstacle is present, a new higher destination is determined until the virtual obstacle is no longer detected. This arrangement causes the falling text displayed on the screen to be selectively “caught” by the user and otherwise manipulated in the virtual environment.

This patent application claims priority under 35 U.S.C. §119 ofprovisional patent application, Ser. No.: 60/178,228, filed Jan. 26,2000, which is hereby incorporated by reference as if set forth in itsentirety herein (including the source code).

COMPUTER PROGRAM LISTING APPENDIX

This application incorporates by reference in its entirety a computerprogram listing appendix containing the source code-listings which weredescribed in the patent application at the time of filing and which havenow been reproduced on compact disk. The compact disk contains the fileentitled “APPENDIX (SOURCE CODE),” was created on Jun. 18, 2003, andencompasses 812 kilobytes.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to interactive computer systems, and inparticular to a method and apparatus for facilitating wireless,full-body interaction between a human participant and acomputer-generated graphical environment including textual data.

2. Description of Related Art

So-called “virtual reality” (“VR”) systems enable users to experiencecomputer-generated environments instead of merely interacting with themover a display screen. Such systems typically require the user to dongoggles, through which he or she perceives the virtual environment, aswell as sensors that encode the user's gestures as electrical signals.The user reacts naturally to the changing virtual environment,generating signals that the computer interprets to determine the stateand progress of the presented environment.

In order to encode a sufficiently board spectrum of gestures tofacilitate natural interaction, VR systems ordinarily require the userto wear, in addition to the goggles, at least one “data glove” to detecthand and finger movements, and possibly a helmet to detect headmovements. Full-body systems, which encode movements from numerousanatomical sites to develop a complete computational representation ofthe user's overall body action, require many more sensors; however, suchsystems, would be capable of projecting the user fully into the virtualenvironment, providing the user with greater control and a heightenedsense of participation ideally suited to interactive simulations.

Unfortunately, numerous practical difficulties limit the capacity ofcurrent VR systems to achieve this goal. The nature of the interactioncurrently offered, even with full-body sensor arrays, is limited. Thecomputational demands placed on a system receiving signals from manysensors can easily overwhelm even large computers, resulting in erratic“jumps” in the visual presentation that reflect processing delays.Moreover, no matter how many sensors surround the user, that cannot“see” the user, and therefore cannot integrate the user's true visualimage into the virtual environment.

Economic and convenience factors also limit sensor-type VR systems. Asthe capabilities of VR systems increase, so do the cost, awkwardness andinconvenience of the sensor array. The sensors add weight and heft,impeding the very motions they are intended to detect. They must alsoordinarily be connected, by means of wires, directly to the computer,further limiting the user's movement and complicating equipmentarrangements.

In order to overcome the limitations associated with sensor-based VRsystems, researchers have devised techniques to introduce the user'srecorded image into a virtual environment. The resulting composite imageis projected in a manner so that it may be viewed by the user, enablingthe user to observe his or her appearance in and interaction with thevirtual environment.

Three such approaches include the VideoPlace system (see, e.g., M.Krueger, Artificial Reality II (1991) and U.S. Pat. No. 4,843,568), theMandala system (see, e.g., Mandala VR News, Fall/Winter 1993; Vincent,“Mandale: Virtual Village” and Stanfel, “Mandela: Virtual Cities,”Proceedings of ACM SIGGRAPH 1993 at 207-208 (1993)), and MIT's systemdisclosed in U.S. Pat. No. 5,563,988. Unfortunately, these systemsexhibit various limitations. For example, Krueger's VideoPlace requiresa special background and ultraviolet lamps, and extracts and representsonly the user's silhouette. The Mandala system can integrate the user'sfull image within the virtual environment it creates, but requires achroma-key blue background. The system developed by MIT provides a3-Dimensional spatial video interaction with a user and is directed todetecting and analyzing gestural information of the user to causepredetermined responses from graphically generated objects within thevirtual environment (e.g., if a virtual user pets a graphicallygenerated dog within the virtual environment, the dog may wag its tail).Regardless of the systems, all of the above-described VR systems employcomplex video analysis and tracking techniques to detect and follow theboundaries of the user's image within the virtual environment.

The disclosure of U.S. Pat. Nos. 5,563,988 of Maes et al, and 4,843,568of Krueger et al. are hereby incorporated by reference as if set forthin their entirety herein.

Applicant has recognized the need for a simple 2-dimensionalvideo-generated real-time interaction system wherein text data is easilyintroduced into a virtual environment, as viewed on a monitor or on aprojection screen. The inputted text data is manipulated in the virtualenvironment by video representation of a human participant (a “virtualuser”). Through movements in the real world, the human participant orvirtual user freely interacts with the inputted text data in the virtualenvironment as the human participant (and other people watching themonitor or projection screen) absorbs the teachings of the text data ashe or she interacts with the text data in the virtual environment.

OBJECTS OF THE INVENTION

It is an object of the present invention to provide a wireless VR systemthat easily accepts text data from a text input to be displayed onto amonitor (or projection screen) so that the font, size, shape, and/ormovements of words and/or characters (e.g, letters, symbols and numbers)of the inputted text data may be altered by a virtual user (a real-timevideo representation of a human participant).

It is another object of the invention to enable the virtual user tointeract with computer-generated, visually represented text data in sucha manner that allows the inputted text to be readable within the virtualenvironment so that the virtual user may learn, understand, or otherwiseexperience the meaning of the textual data as he or she interacts withthe text within the virtual environment.

It is another object of the invention to provide an interaction displaysystem wherein text data and the video image of a user is combinedwithin a virtual environment to promote marketing, advertisement, sales,and entertainment.

It is another object of the invention to provide an interaction displaysystem wherein text data and the video image of a user is combinedwithin a virtual environment and wherein the text data viewed within thevirtual environment may be changed in realtime in response to actions bythe user or by others.

It is another object of the invention to provide an interaction displaysystem wherein text data and the video image of a user is combinedwithin a virtual environment and wherein the text data is imported froma local text file.

It is another object of the invention to provide an interaction displaysystem wherein text data and the video image of a user is combinedwithin a virtual environment and wherein the text data is imported froma remote site over the Internet.

It is another object of the invention to provide an interaction displaysystem wherein text data and the video image of a user is combinedwithin a virtual environment and wherein the text data is inputteddirected from a keyboard.

The foregoing and other objects of this invention can be appreciatedfrom the Summary of the Invention, the drawing Figures and DetailedDescription.

SUMMARY OF THE INVENTION

In accordance with the invention, a controlling software programinstructs an interactive display system. The interactive display systemincludes an electronic camera (e.g., charge-coupled device or “CCD”)which intermittently records (in realtime) a human participant's (auser) image and inputs the video image into a computer. The computercreates a digitized version of the inputted image (e.g., a bitmap ofpixels) and thereby creates a “virtual user image”. According to oneembodiment, the camera is positioned behind a large viewing screen whichfaces the user and records the user's image through an opening providedwithin the screen. The computer further receives a data record includingtext data and text controlling parameters, preferably including at leasta fall rate value. The data record may be supplied to the computer usingany of several conventional memory mediums, such as a hard drive memory,disc memory, compact disc, or the data record may only include a textstring which is supplied to the computer directly from the Internetthrough a connected modem, for example, or through the use of aconnected keyboard. Alternately, the text may be inputted to thecomputer directly from a word processing program, such as Corel'sWordPerfect®, or Microsoft Word® as a text file or by “copying” selectedtext from an open text file directly into the present software program.

According to the invention, upon receiving a captured video image of theuser, the software program (see appendix for source code) first parsesthe text data into lines of text. The program determines which line orlines of text should be displayed on the screen and further determinesthe destination of each character within any line that is to bedisplayed on the screen, according to the fall rate assigned to eachcharacter. The program then compares the pixel color value of“destination pixels” (pixels located at the destination of eachparticular character) with a threshold color value. If the color valueof the destination pixels is below the threshold value (i.e., lighter),then the particular character will be positioned at a destinationaccording to its prescribed fall rate. If, however, the color value ofthe destination pixels exceeds the threshold value (i.e., darker), thenit is determined that the virtual user's image is located at thedestination of the particular character of the line of text. In suchinstance, the program locates a suitable new destination positionedvertically above the original destination which is lighter in colorvalue than the threshold value.

Once all the characters of all the lines of text intended to beprojected onto the screen are provided with suitable destinations, theprogram instructs the computer to display all of the characters on thescreen at their respective destinations with the particular capturedvideo frame. The process continues where a new image frame is availablefrom the camera.

The program analyzes each captured image as it acquires at a prescribedsampling rate. The sampling rate is dependent on the speed of the videocard components and drivers, the speed of the computer, and lightsensitivity of the digital camera.

According to the invention, as the individual characters of a singleline make their way down along the projection screen, the program keepstrack of when a predetermined number of characters (e.g., at least halfof them) travels a predetermined distance from the top of the viewingscreen (such as the halfway point). If this is the case, the programcauses the entire line of text to fade at a prescribed fade rate. Whenone line fades at the lower portion of the screen, another line appearsat the top of the screen, with its characters beginning to fall. It ispreferred that at least three lines of text be displayed on the screenat any given time.

Another aspect of the invention displays each line of text data as aseparate color so that viewers may easily discern the letters orcharacters from different lines of inputted text, allowing the lines oftext to be read relatively easily, as the letters fall towards and reston various darker elements located within the virtual environment, suchas the arm of a virtual user and perhaps a “virtual umbrella image”carried by the virtual user.

According to another aspect of the invention, each character of aninputted text string is horizontally held so that each character isrestricted to vertically movement as it descends on the viewing screenat its prescribed fall rate or ascends upwardly with similar upwardmovement by the virtual user.

According to another embodiment of the invention, the program generatesa shadow zone of dark pixels which are projected onto the viewingscreen. The shadow zone is aligned with the optical input of the digitalcamera so that the camera will be protected from the relatively brightlight of the projector. The computer generated shadow zone helps preventthe images recorded by the camera from becoming “washed out” by thelight from the projector shining on the camera's lens.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing discussion will be understood more readily from thefollowing detailed description of the invention, when taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a schematic illustrating an interactive display systemincluding a projection screen, a pickup camera, a projector, a computer,and a text input, according to the invention;

FIG. 2 is a perspective view of the inventive display system of FIG. 1,showing the field of view of the camera, according to the invention;

FIG. 3 is a block diagram of the interactive display system, accordingto the invention;

FIG. 4 is a illustration of an exemplary text string to be inputted intothe interactive display system, according to the invention;

FIG. 5 is a illustration of an exemplary data record which includes thetext string of FIG. 4 and further includes text-controlling parameters,according to the invention;

FIG. 6 is a flow diagram showing steps of obtaining and combining videoand text data, according to the invention;

FIG. 7 is a front view of the viewing screen, illustrating an exemplarydisplayed virtual image including a virtual user holding a virtualumbrella, and further including falling text, according to theinvention;

FIG. 8 is an enlarged view of a portion of the virtual umbrella of FIG.7 and select letters of the text, according to the invention; and

FIG. 9 is a enlarged partial front view of the viewing screen showingdetails of a central opening and a computer generated shadow zone,according to the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIGS. 1, 2, and 3 the interactive display system 10,according to the present invention, is shown including a large displayscreen 12 having a generally central and small opening 14, an imageprojector 16 positioned to project an image (along projection beam 17)onto the front surface of screen 12, and a camera 18, preferably a miniblack and white CMOS-type video camera (alternately a ConnectrixQuickCam® by Logitech® with appropriate platform drivers may be used).The camera 18 is positioned behind screen 12 with its image-receivinglens aligned with opening 14, so that a person (a user) standing infront of screen 12 (in the area represented by reference numeral 15, inFIGS. 1 and 2) and within the field of view 19 of the camera 18, isrecorded by the camera. The camera 18 is preferably sufficiently lightsensitive to “capture” repetitive images of the user in apparentrealtime (i.e., generating a sufficient number of consecutive images sothat when the sampled images are eventually projected, the motions ofthe user appear smooth and continuous). It is preferred, although notshown, that a rear surface or wall be located opposite the screen 12,behind the user, and that this rear surface be white and illuminated.Although the rear surface is not necessary for the present system tooperate, it does allow the present program to quickly and easilydistinguish the virtual user from the background.

The present interactive display system 10 further includes a computer 20having a memory 21 and a text input 22, as shown in FIG. 1. Computer 20may be conventional, and preferably includes a fast central processingunit (CPU), such as Intel's Pentium II® or higher. The video card of thecomputer preferably includes enough VRAM to display 640 X 480 resolutionat high color (16 bit). If a CMOS type camera 18 is being used behindthe large screen 12, then the computer 20 further requires a videocapture card 24 to create a bitmap of the analog image of the CMOScamera 18. If a Connectix®-type camera is used (or equivalent), nocapture card 24 is necessary because the Connectix®E-type camera usessoftware to digitize a recorded image. The computer 20 preferably usesMicrosoft's Window 98® or Microsoft's Windows NT® operating systems.

As is understood by those skilled in the art, the video capture card 24is used to receive analog image data from the camera 18 and digitize theimage by generating a bitmap (the present program can control the rateof sampling). During each sampling, as described below, the video card24 captures a frame of image data from camera 18 as the camera recordsthe user and digitizes the frame into array of picture elements, or“pixels,” called a bitmap. Depending on the format, each pixel within abitmap is identified by an RGB color code (a three number code whichidentifies the exact color of the pixel). The bitmap representing thereceived frame of analog image data is stored in memory for analysis andmanipulation by the software program.

The computer receives a data record including text data andtext-controlling parameters from text input 22. As described below thepresent software program manipulates both the digitized image of thereceived frame of analog image data and the text data from text input 22and combines the video and text data onto the projection screen 12 in agenerally smooth and continuous manner.

Text input 22 may include any of a variety of conventional inputdevices, such as a hard drive memory, disc memory, compact disc, or aconventional Internet connection (such as through a connected modem), orthrough the use of a connected keyboard. Alternately, the text may besupplied to the computer directly from a word processing program, suchas Corel's WordPerfect®, or Microsoft Word® as a text file or by copyingselected text from an open text file directly into the program. Textdata is preferably inputted into the memory 21 of computer 20 for laterretrieval by the present program and displayed in accordance with theinvention, or alternately, the text is inputted in realtime forimmediate projection onto screen 12.

An example of text data 30 for use with the present invention is shownin FIG. 4 (shown displayed on a computer monitor 32): “When it rains, itpours. When it stops, its wet.” This exemplary text data 30 is inputtedinto the computer 20 by any conventional means and is parsed intoappropriately-sized lines 31 (depending on the size of the projectionscreen 12). If the text data 30 exceeds a certain number of characters36, the program will parse the text data 30 into separate lines of text31 so that each line may fit onto the screen 12 (the wider the screen12, the longer each line of text 31). The program will form a line datarecord to keep track of which lines of text 31 are on the screen 12, arefading from the screen 12, and should be introduced at the top of thescreen. Each line of text 31 located on the line data record will beintroduced in a prearranged order and every line of text 31 which isbeing projected on the screen 12 will be electronically marked orflagged in the line data record. In this manner, as one line of text 31fades from the screen 12, its electronic mark is removed from the linedata record and a subsequent line of text 31 will be introduced onto thescreen 12.

Each line located in the line data record, is also stored in a characterdata record 34, shown in FIG. 5. According to one aspect of theinvention, the character data record 34 may further include variousparameters which will control how each character 36 of each line of text31 moves and appears within the virtual environment, as viewed on screen12. The parameters may vary depending on the desired effect of thecharacters 36 of each line of text 31 within the virtual environment anddepending on the complexity and speed of the interaction system.

The controlling parameters may include a fall rate value which controlsthe speed each character 36 of each line of text 31 appears to fall onthe screen 12. The fall rate value may be defined in a variety of ways,such as the number of pixels traveled in one second or between capturedframes of video.

Another character parameter may be a fall delay value which determinesthe length of time each character 36 will remain at the top of thescreen 12 prior to falling.

A color code may be assigned to each character 36 or all of thecharacters within each line of text 31.

Another parameter for each character 36 is a horizontal hold conditionwhich, when set, restricts each character to vertical movement whenprojected within the virtual environment.

A fade parameter controls whether a character 36 fades when reaching thebottom of the screen 12 within the virtual environment.

An edge-detect parameter determines whether a character will recognizethe virtual image of a user within the virtual environment (i.e.,determines if a virtual user interacts with the falling characters).

The distance the characters 36 “fall” along the screen 12 within thevirtual environment may also be controlled using a maximum fall distanceparameter.

The above-listed character parameters are provided to illustrate thateach character 36 may be controlled in a variety of ways independentlyof the other characters 36 within a line of text 31. Although a varietyof parameters are possible, as exemplified above, it is preferred thatmost, if not all, include default values which simplify the setup anduse of the present program. It is preferred that an operator merely berequired to input the text data 30 into computer 20 without furthersetup.

In operation, referring to FIG. 6, the present program first receivesinput text data 30 at step 100. The text data 30 is first divided upinto lines of text 31 at step 110. The number of characters 36 in eachline of text 31 depends, in part, on the dimensions of the viewingscreen 12. The lines of text 31 may be divided into any number ofcharacters, following, for example, the length of each line of a poem. Afall rate value (and perhaps other variables) is assigned to eachcharacter 36 of each line of text 31 at step 120. As described above, itis preferred that all the characters 36 of any given line of text 31have the same fall rate. At step 130, a frame of video from the camerais received and digitized by the video capture card 24, generating abitmap wherein an RGB color value is assigned to each pixel defined inthe captured image frame. Once the program starts capturing frames ofvideo data, the program enters into a callback mode whereby every time anew frame of video data is received, select pixels of the bitmap areanalyzed, as described below. Since only select pixels are analyzed bythe present program, the speed of the program is much faster than priorart VR systems which rely on analyzing each pixel in the bitmap for eachframe of image data received.

At step 140, the program determines which lines of text 31 are to bedisplayed on the screen 12. It is preferred that three lines of text 31be displayed on the screen 12 at any given time. The number of lines oftext 31 may vary, of course, depending on the particular application,the size of the viewing screen 12, and the desired effect. At step 150,in FIG. 6, the program determines the destination for a selectedcharacter 36 among the characters of the “to be displayed” lines of text31, according to the predetermined fall rate of the selected character36. At step 160, the color value of the “destination pixels” or thosepixels which are located at the destination of the selected character 36at step 150. The program, at step 170, then compares the read colorvalue of the destination pixels with a threshold color (or contrast)value. At step 180, if it is determined at the comparison step 80 thatthe destination pixel color value is greater than the threshold colorvalue, then the destination of the selected character 36 is raised by apredetermined number of pixels, preferably a single pixel, at step 190.In this instance, the pixel color value at the newly determineddestination of the selected character 36 is again read at step 160, itsvalue compared again with the threshold value at step 170 and a decisionis again made at step 180 until the read pixel value of the destinationpixel is less than or equal to the threshold color value.

At this point, the determined destination of the selected character 36is stored in memory at step 200. At step 210, it is determined if anymore characters 36 remain within the “to be displayed” lines of text 31.If there are more characters 36 remaining within the “to be displayed”lines of text 31, another character 36 is selected at step 220 and againthe destination of the newly selected character 36 is determined at step150 and the above-described process is repeated until the destination ofeach character 36 of the “to be displayed” lines of text 31 have beendetermined. When this occurs, the captured video image frame togetherwith each character 36 of the “to be displayed” lines of text 31 aredisplayed onto screen 12 (at step 230), with each character 36 locatedat each respective destination, as determined by the program and asstored at step 200.

The program then returns to step 130 to acquire another image frame ofvideo from camera 18 and repeats the above described steps, as shown inthe flow diagram of FIG. 6.

The end result is that the individual characters 36 of their respective“to be displayed” lines of text 31 will appear to fall (at theirprescribed fall rate) from the top of the projection screen 12, asillustrated in FIGS. 7 and 8. In FIG. 7, an exemplary projected image onscreen 12 is shown to have a shoulder 240, and an arm 242 of a virtualuser 244. The actual user (not shown) and the virtual user 244 areholding an umbrella 246 which appears on the screen 12 in this example,as a virtual umbrella 246. As is shown, the program has displayed thevideo image of the virtual user (shown in part) 244 holding the virtualumbrella 246 and the inputted lines of text 31: “When it rains, itpours”. In this exemplary scene, a new line of text 31: “When it rains,it pours” has just been introduced at the top portion of the screen 12,generally indicated by arrow 248 of FIG. 7. The characters “h” and “e”of the word “When” have already started to fall at their quicker fallrates. Various characters 36 from previously introduced lines of text 31(in this example, all the lines of text 31 shown on the screen 12 arethe same: “When it rains, it pours”) have been drifting downwardly fromthe top portion 248 of the screen 12 to towards the bottom portion ofthe screen, generally indicated by arrow 250 in FIG. 7.

Following the steps described above, as the computer 20 determines thedestination of each character 36, it quickly reads the pixel colorvalues at that destination of the captured image and determines whethera virtual obstacle is already located there. If no obstacle is there(i.e., the read pixels are less than or equal to a threshold value),then the particular character 36 appears on the screen 12 at thedestination and, as subsequent images are displayed on screen 12, theparticular character 36 will appear to slowly fall, like snow. Ifhowever, a virtual obstacle is determined to be located at thedestination of a particular character 36, then a new destination locatedhigher on the screen 12 is selected so that the character 36 appears tointeract with the virtual obstacle located in the virtual environment.This interaction is illustrated in FIG. 8, wherein an enlarged partialsection of the umbrella 246 is shown “capturing” the characters “r”,“a”, and “i” of the word “rains”. These characters 36 will remain at thefirst “light” pixel (below th pixel color threshold value) located abovethe relatively dark pixels which make up the umbrella 246 so that thecharacters 36 appear to have fallen onto and are supported by theumbrella 246. If the user raises the umbrella 246, the program willquickly detect this upward movement because the “lighter” pixel betweenthe character 36 and the umbrella 246 will become a dark value causingthe computer to respond by raising the affected characters 36 upwardlywith the umbrella movement, so that the characters 36 appear to havebeen lifted by the umbrella 246. The characters 36 can move upwardlyvery quickly, but will preferably always fall at the predetermined fallrate, as discussed above.

As subsequent video images are sampled, each character 36 of a line oftext 31 appear to gradually descend on the screen 12 until the character36 confronts a dark pixel, signifying a virtual obstacle within thevirtual environment. Although the present program may be setup so thatthe characters 36 fall at any fall rate, it is preferred that thecharacters 36 fall in a manner which is somewhat familiar and pleasingto the user, similar, for example, to the graceful movements of anairborne leaf slowing falling from a tree. A user can “catch” thefalling leaf and quickly push it back into the wind currents, but theleaf will always fall back down at its leisurely pace. By “training” theintroduced characters 36 to respond in this manner within the virtualenvironment, the user is likely to remain comfortable and will beencouraged to further interact and explore within the virtualenvironment.

The text data 30 may be the lines of a poem, song, or even the tag-lineof an advertisement of a particular product and may be projected in anysize, shape or font (system fonts are preferably used so that theprocessing is faster). As the user interacts with the introduced linesof text 31, he or she must physically manipulate the falling characters36 and perhaps capture several of them to read them and understand theactual text, but the fun of combining such a physical activity withreading only encourages the user to succeed in reading the textualinformation. Whatever message the textual information is to convey ismore likely to reach the viewer or user when the user gets toparticipate with the message in an interactive game-like environment.

The textual information introduced into the present program may becontinuously changed, such as in the case of weather reports, or newsstories. Alternately, the textual information may be altered inrealtime, perhaps in response to the particular actions of the user. Inthis instance, a user may be asked questions in a text format with thequestion appearing on the screen 12 together with the image of the user.When the user reads the question on the screen 12, perhaps aftercapturing the characters 36, he or she can audibly answer the question.The response to the question can be heard by the operator of the system.who may then ask another question (again in text format) which relatesor corresponds to the user's response to the previous question. Thistype of interaction between the text-input operator and the user mayfurther entertain and teach the user and any viewer of the system.

According to another aspect of the invention, text data may beintroduced one line at a time allowing a first line of text 31 to beintroduced at the top of the of the viewing screen 12 so that eachcharacter 36 begins to fall (as viewed on the viewing screen 12), eitherat similar fall rates or at different ones. When a predetermined numberof falling characters 36 (e.g., at least half of them) travels apredetermined distance from the top of the viewing screen 12, a secondline (or select characters of the second line) of text 31 will appear atthe top of the screen 12. The characters 36 of the newly introducedsecond line of text 31 will begin to fall along the viewing screen 12 ina similar manner to those of the first line of text. Third and otherlines of text 31 may be displayed in a similar and repeating manner.

Another aspect of the invention causes inputted lines of text 31 to fade(at a prescribed fade rate) when a predetermined number of characters(e.g., at least half of them) reach a predetermined position on theviewing screen (e.g., enter the lower half of the screen). In apreferred embodiment, at any given moment, three lines of text 31 aredisplayed on the viewing screen 12, one line of text 31 being introducedat the top of the screen, a second one falling in the middle of thescreen (an “interactive region”), and a third line of text 31 fading outat the lower portion of the screen. Of course, some of the characters 36(or letters) of earlier introduced lines of text 31 may remain on thescreen, “captured” by virtual obstacles, such as the above-describedumbrella 246 of FIG. 7. If one particular character 36 from a first lineof text 31 remains held in vertical space by a virtual obstacle (e.g.,umbrella 246), the same character 36 from subsequent lines of text 31will eventually overlay the first “held” character 36 so that similarcharacters 36 held by virtual obstacles will not accumulate in numberover time. However, it is also contemplated that as characters 36descend onto each other on the screen 12, the program can be altered sothat any higher character 36 will consider a lower “held” character tobe part of the virtual obstacle so that the characters will pile up(stack on top of each other), until the obstacle is cleared (i.e., theuser moves).

According to another aspect of the invention, each line of text 31 isdisplayed as a separate color. In this arrangement, viewers may easilydiscern the characters from different lines of text 31 so that the linesof text 31 to be read relatively easily even as the individualcharacters 36 fall towards and rest on various darker elements locatedwithin the virtual environment, such as the arm of a virtual user andperhaps a “virtual umbrella image” carried by the virtual user.

According to another aspect of the invention each character 36 ofinputted text data 30 is horizontally-held, so that each character 36 isrestricted to vertically movement as it descends on the viewing screen12 at its prescribed fall rate, or ascends with upward movement of avirtual obstacle.

The techniques described herein for integrating a user's image andinputted textual data within a digitally represented environment can beapplied to numerous applications including fun and educationalcommunication applications for use in schools to help kids learn to reador just have fun. In this application, Kids could be asked, for example,to interact with a falling sentence by “capturing” a verb located withinthe falling sentence.

The present system can also be installed at a store front, in which casea display monitor may be positioned within the store so that it may beviewed by passing pedestrians. In this application, the appearance of apedestrian on the monitor will invariably attract the attention of thepedestrian, who will stop and interact with the textual message fallingall around his or her image being displayed on the screen. The message,in this example, would help lure customers inside the store to purchaseproducts.

Other useful applications for this technology include performance art,either used in connection with an artist or an audience member. Thepresent invention can be setup within an art gallery, exhibit, or museumas an interesting and alluring way to convey any textual informationeither as a piece of art work, or as a source of information, such as aninteractive information kiosk.

According to another aspect of the invention, the present software isconnected to the Internet through appropriate modems, T1, or DSLcommunication lines for the purpose of allowing Internet users tointroduce text data 30 to a particular interactive display shown overthe Internet. Also, the software program may be used with a local PCwhereby the computer's monitor becomes the viewing screen, and a local“webcam” type camera (or any other appropriate camera) may be used torecord the user, who in this case, would be positioned in front of thecomputer. If a CRT or LCD type monitor is used as the screen 12, thecamera may be positioned on top of (or adjacent to) the monitor.

Referring to FIG. 9, an enlarged partial screen 12 showing the centralopening 14 is shown, according to another embodiment of the invention.Applicant has discovered that light from projector 16 directed to theoptical input lens of camera 18 causes the images captured by the camerato appear “washed out” because the camera automatically compensates forcorrect exposure. To overcome this problem, the present programgenerates a shadow zone 13 of dark pixels which are projected onto theviewing screen 12. The shadow zone 13 is aligned with the optical inputof the digital camera so that the camera will be protected from therelatively bright light of the projector. The computer-generated shadowzone 13 helps prevent the images recorded by the camera from becoming“washed out” by the light from the projector shining on the camera'slens. The shadow zone 13 is shown as a darkened square, however, theshadow zone 13 may take on any useful shape to cover opening 14, andalso may be any shade of gray. The program includes means to project theshadow zone 13 when each image and lines of text 31 are projected ontoscreen 12. Also, the projected position of the shadow zone 13 on screen12 may be easily moved by the program, using an appropriate inputdevice, such as a mouse or a keyboard (not shown).

The present software program (see appendix for actual source code) iswritten in C++ programming language using standard Microsoft® andVideo-for-Windows® Libraries.

Although the above detailed description of the invention assumes thepresence of a single user, the invention is capable of simultaneouslysupporting multiple users. These users can occupy the same space infront of a single camera 18, or images of each user can be inputted to asingle computer 20 using separate cameras at physically disparatelocations. In the latter case, remote image data reaches the inventionby means of suitable network connections, and the invention sendspicture signals to the user's remote monitor over the same facility. Inthis way, all users interact in the same virtual world.

The terms and expressions employed herein are used as terms ofdescription and not of limitation, and there is no intention, in the useof such terms and expressions, of excluding any equivalents of thefeatures shown and described or portions thereof, but it is recognizedthat various modifications are possible within the scope of theinvention claimed.

1. A method for facilitating real-time interaction between a user and adigitally represented visual environment within which the user's movingimage is integrated, said method including the use of a computer,electronic memory, a display, a video camera, and a video input device,the method comprising the steps of: storing a first computer generateddigital image in said electronic memory; assigning a velocity ofmovement to said digital image, said velocity of movement including arate of movement and a direction of movement of said digital image;recording the image of said user using said video camera; simultaneouslydisplaying the image of said user and said stored first digital imageonto said display, at a predetermined refresh rate; digitallyrepositioning said displayed first digital image on said displayaccording to said assigned velocity of movement; comparing the relativeposition of said displayed image of said user and said displayed firstdigital image; determining when said displayed first digital image andthe displayed image of said user are within a predetermined distance onsaid display; changing said velocity of movement of said displayed firstdigital image in response to determining that said displayed firstdigital image and the displayed image of said user are within saidpredetermined distance; and simultaneously displaying said first digitalimage at said new velocity of movement, and the image of said user.
 2. Amethod for facilitating real-time interaction between a user anddigitally represented text data on a display within which the user imageis integrated, the method comprising the steps of: storing text data insaid electronic memory; assigning a velocity of movement to said textdata, said velocity of movement including a rate of movement and adirection of movement of said text data; recording the image of saiduser using a video camera; simultaneously displaying the image of saiduser and said stored text data onto said display, at a predeterminedrefresh rate; digitally repositioning said displayed text data on saiddisplay by said assigned velocity of movement; comparing the relativeposition of said displayed image of said user and said displayed textdata; determining when said displayed text data and the displayed imageof said user are within a predetermined distance on said display;changing the velocity of movement of said displayed text data inresponse to determining that said displayed text data and the displayedimage of said user are within said predetermined distance; andsimultaneously displaying said text data at said new velocity ofmovement and the image of said user.
 3. The method of claim 2, whereinsaid display includes an upper edge and a lower edge, and said velocityof movement of said text data includes a direction towards said loweredge of the display.
 4. The method of claim 2, wherein said velocity ofmovement of said text data includes a first rate of movement, and saidnew velocity of movement of said text data includes a second rate ofmovement.
 5. The method of claim 2, wherein said new velocity ofmovement of said text data includes no movement.
 6. The method of claim2, wherein said assigned velocity of movement of said text data includesa first direction of movement, and said new velocity of movement of saidtext data includes a second direction of movement, said second directionof movement being opposite said first direction of movement.
 7. Themethod of claim 2, wherein said text data is initially displayed on saiddisplay at a predetermined location prior to moving at said assignedvelocity.
 8. The method of claim 2, wherein prior to performing the stepof storing text data into electronic memory, a step of receiving textdata from a keyboard is performed.
 9. A method for facilitatingreal-time interaction between a user and digitally represented text dataon a display within which the user image is integrated, the methodcomprising the steps of: storing text data in an electronic memory;assigning a velocity of movement to said text data, said velocity ofmovement including a rate of movement and a direction of movement ofsaid text data; storing a threshold pixel color value in electronicmemory; recording the image of said user using a video camera;simultaneously displaying the image of said user and said stored textdata onto said display, at a predetermined refresh rate, therebycreating a combined image; determining a destination of said text data,according to the assigned velocity, said text data destination being thepoint within the combination image where the text will next bedisplayed; measuring the pixel color value of the display image at thedetermined text data destination; comparing the measured pixel colorvalue at the text data destination with said stored threshold colorvalue; displaying said text data at said text data destination inresponse to said comparing step determining that said measured pixelcolor value at the text data destination is less than said storedthreshold color value; and displaying said text data at a positionwithin said combined image on said display other than said text datadestination in response to determining in said comparing step that saidmeasured pixel color value at the text data destination is greater thanthe stored threshold color value.
 10. The method of claim 9, furthercomprising the step of changing the velocity of the text data inresponse to said comparing step determining that said measured pixelcolor value at the text data destination is greater than the storedthreshold color value.
 11. An arrangement capable of facilitating aninteraction of at least one portion of a physical object with at leastone portion of a virtual object when being executed by a processingarrangement, comprising: a first set of instructions which, whenexecuted by the processing arrangement, is configured to receive firstinformation associated with the at least one portion of a firstrepresentation of the virtual object and second information associatedwith at least one section of a second representation of the at least oneportion of the physical object, wherein the second representation isgenerated irrespective of an implementation of a particularcharacteristic associated with at least one of a color or anillumination of a background associated with the at least one portion ofthe physical object; a second set of instructions which, when executedby the processing arrangement, is configured to determine a spatialrelationship between respective positions associated with the firstrepresentation and the second representation by analyzing the first andsecond information; a third set of instructions which, when executed bythe processing arrangement, is configured to modify at least one portionof the first information based on the determination made by the secondset of instructions; and a fourth set of instructions which, whenexecuted by the processing arrangement, is configured to provide furtherdata associated with the first and second representations based on themodified portion at least one of for display or in a virtual space. 12.The arrangement according to claim 11, wherein the third set ofinstructions is configured to modify the at least one portion of thefirst information by repositioning at least one of the relative positionassociated with at least one section of the first representation of theat least one portion of the virtual object and the respective positionassociated with at least one section of the second representation of theat least one portion of the physical object.
 13. The arrangementaccording to claim 11, wherein the first representation is associatedwith a first digital image of the virtual object.
 14. The arrangementaccording to claim 13, wherein the second representation is associatedwith a second digital image of the at least one portion of the physicalobject.
 15. The arrangement according to claim 14, further comprising afifth set of instructions which, when executed by the processingarrangement, is configured to reposition at least one of the firstdigital image and the second digital image at least one of for displayor in the virtual space.
 16. The arrangement according to claim 15,wherein the fourth set of instructions is capable of providing at leastone of the first repositioned digital image or the second repositioneddigital image at least one of for display or in the virtual space. 17.The arrangement according to claim 16, wherein the first digital imageis capable of being repositioned in a direction relative to the seconddigital image.
 18. The arrangement according to claim 14, wherein thespatial relationship is provided as a function of a determination ofwhen the first digital image at least one of reaches or is positioned ata predetermined distance from the second digital image.
 19. Thearrangement according to claim 11, wherein the second representation isassociated with a digital image for the at least one portion of thephysical object.
 20. The arrangement according to claim 11, wherein thefurther data is associated with data provided for at least one ofmarketing, advertising, sales, entertainment, learning, education,games, conveying weather or conveying news.
 21. The arrangementaccording to claim 11, further comprising a sixth set of instructionswhich, when executed by the processing arrangement, is configured toobtain destination data of the virtual object, the destination databeing related to a particular position which is targeted to be reachedby the virtual object.
 22. The arrangement according to claim 21,wherein the destination data is related to a particular position whichit targeted to be reached by the relative position associated with thefirst representation based on the second information.
 23. Thearrangement according to claim 11, further comprising a seventh set ofinstructions which, when executed by the processing arrangement, isconfigured to obtain third information associated with a speed ofmovement of the virtual object which is associated with the firstrepresentation.
 24. The arrangement according to claim 23, wherein atleast one portion of the third information is capable of being modifiedbased on the determination by the seventh set of instructions.
 25. Thearrangement according to claim 11, further comprising an eighth set ofinstructions which, when executed by the processing arrangement, isconfigured to modify a speed of a movement of the relative positionassociated with the virtual object as a function of the secondrepresentation.
 26. The arrangement according to claim 11, furthercomprising a tenth set of instructions which, when executed by theprocessing arrangement, is configured to reposition at least one of therelative position associated with at least one section of the firstrepresentation of the at least one portion of the virtual object and therespective position association with at least one section of the secondrepresentation of the at least one portion of the physical object.
 27. Amethod for facilitating an interaction of at least one portion of aphysical object with at least one portion of a virtual object,comprising: receiving first information associated with the at least oneportion of a first representation of the virtual object and secondinformation associated with at least one section of a secondrepresentation of the at least one portion of the physical object,wherein the second representation is generated irrespective of animplementation of a particular characteristic associated with at leastone of a color or an illumination of a background associated with the atleast one portion of the physical object; determining a spatialrelationship between respective positions associated with the firstrepresentation and the second representation by analyzing the first andsecond information; modifying at least one portion of the first databased on the determination of the special relationship; and providingfurther data associated with the first and second representations basedon the modified portion at least one of for display or in a virtualspace.
 28. A system for facilitating an interaction of at least oneportion of a physical object with at least one portion of a virtualobject, comprising: a first arrangement configured to receive firstinformation associated with the at least one portion of a firstrepresentation of the virtual object and second information associatedwith at least one section of a second representation of the at least oneportion of the physical object, wherein the second representation isgenerated irrespective of an implementation of a particularcharacteristic associated with at least one of a color or anillumination of a background associated with the at least one portion ofthe physical object; a second arrangement configured to determine aspatial relationship between respective positions associated with thefirst representation and the second representation by analyzing thefirst and second information; a third arrangement configured to modifyat least one portion of the first data based on the determination madeby the determination made by the second arrangement; and a fourtharrangement configured to provide further data associated with the firstand second representations based on the modified portion at least onefor display or in a virtual space.
 29. An arrangement capable offacilitating an interaction of at least one portion of a physical objectwith at least one portion of a virtual object when being executed by aprocessing arrangement, comprising: a first set of instructions which,when executed by the processing arrangement, is configured to receivefirst information associated with the at least one portion of a firstrepresentation of the virtual object and second information associatedwith at least one section of a representation of the at least oneportion of the physical object; a second set of instructions which, whenexecuted by the processing arrangement, is configured to obtaindestination data associated with the virtual object, the destinationdata being related to a particular position which is targeted to bereached by the virtual object; a third set of instructions which, whenexecuted by the processing arrangement, is configured to determinewhether the destination data is at least one of reached or configured tobe reached by the representation; a fourth set of instructions which,when executed by the processing arrangement, is configured to modify atleast one portion of the first information based on the determination;and a fifth set of instructions which, when executed by the processingarrangement, is configured to provide the modified first information andthe second information at least one of for display or in a virtualspace.
 30. The arrangement according to claim 29, wherein the firstrepresentation is associated with a first digital image of the virtualobject, and the second representation is associated with a seconddigital image of the at least one portion of the physical object. 31.The arrangement according to claim 29, wherein the destination data isdetermined based on the second information.
 32. An arrangement capableof facilitating an interaction of at least one portion of a physicalobject with at least one portion of a virtual object when being executedby a processing arrangement, comprising: a first set of instructionswhich, when executed by the processing arrangement, is configured toreceive first information associated with the at least one portion of afirst representation of the virtual object and second informationassociated with at least one section of a second representation of theat least one portion of the physical object; a second set ofinstructions which, when executed by the processing arrangement, isconfigured to obtain third information associated with a speed ofmovement of the virtual object; a third set of instructions which, whenexecuted by the processing arrangement, is configured to determinewhether a relative position associated with the first representation atleast one of reaches, is configured to reach or is provided at apredetermined distance from a relative position associated with thesecond representation by analyzing the first and second information; afourth set of instructions which, when executed by the processingarrangement, is configured to modify at least one portion of the thirdinformation based on the determination; a fifth set of instructionswhich, when executed by the processing arrangement, is configured toreposition the relative position of the first representation as afunction of the at least one modified portion of the third information;and a sixth set of instructions which, when executed by the processingarrangement, is configured to provide the first modified representationand the second representation at least one of for display or in avirtual space.
 33. The arrangement according to claim 31, wherein thefirst representation is associated with a first digital image of thevirtual object, and the second representation is associated with asecond digital image the at least one portion of the physical object.